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DIGITAL CAMERAS WITH LUMINANCE CORRECTION 



TECHNICAL FIELD 

[0001] The present invention generally relates to image processing, and 

more particularly, to digital cameras with luminance correction. 

BACKGROUND 

[0002] When photographing a scene, light rays emitted from objects within 

the scene are recorded on a film such as regular film or digital film. Hence, 
photography involves the recording of these light rays. When lighting conditions 
are improper (e.g., when photographing in low light), pictures lack some of the 
scene information when compared with pictures taken in sufficient lighting 
conditions. 

[0003] Taking satisfactory photos under dim lighting conditions has 

historically posed a very difficult problem. Often, the images are blurred and/or 
underexposed. Underexposures generally results from not exposing the film to 
sufficient amounts of light. Underexposure may be somewhat corrected by 
exposing the film for a longer period, for example, by using a lower shutter speed 
to keep the shutter open for a longer period. Lower shutter speed, however, results 
in blurring. The blurring problem is exasperated when using a hand-held camera 
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(e.g., rather than a tripod), in part, because of the increased movement during 
shutter openings. Blurring may also occur due to movement of the objects within 
the scene during shutter openings. 

[0004] A couple of common solutions include use of flashes (to compensate 

for low lighting by introducing additional lighting) or a film with higher ISO (a 
prefix set by the International Organization for Standardization). 

[0005] Using flashes is limiting for a variety of reasons. For example, 

flashes are only operational over relatively short distances. Also, flashes may 
result in change of colors, yielding an inaccurate representation of the scene. 
Multiple flashes (e.g., with remote activation) may be utilized to improve the 
results of flash photography, but setting up several flashes around a scene may not 
always be feasible (e.g., in outdoor photography or when capturing shots with 
short notice). 

[0006] Higher ISO film is also limiting for a number of reasons. In 

traditional photography, the film is often only changeable one roll at a time. 
Accordingly, when a camera is loaded with higher ISO film (e.g., suitable for low 
lighting conditions), the camera can not be used for normal lighting conditions 
without limiting the photographers options (e.g., where pictures have to be taken 
at higher shutter speeds to avoid overexposure). In digital photography, the 
performance of higher ISO settings entirely depends on the camera sensor, which 
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can significantly vary between different cameras. Moreover, an even more 
important shortcoming is the relatively higher amount of noise that results from 
using the higher ISO. 

[0007] Currently, there are several techniques for improving the quality of 

blurred images, e.g., resulting from an exposure time above the safe shutter speed. 
Generally, the safe shutter speed is a speed no slower than the reciprocal of the 
focal length of the lens. These techniques can be roughly classified into in-process 
and post-process approaches which limit motion blur due to, for example, a long 
exposure time, camera shake, or object movement. 

[0008] In-process approaches are mainly hardware-based techniques, where 

lens stabilization is achieved by camera shake compensation. Alternatively, high- 
speed digital cameras (such as those with complementary metal oxide 
semiconductor (CMOS) sensors) can perform high-speed frame captures within 
normal exposure time which allows for multiple image-based motion blur 
restoration. The in-process techniques are able to produce relatively clear and crisp 
images, given a reasonable exposure time. However, they require specially 
designed hardware devices. 

[0009] On the other hand, post-process methods can be generally 
considered as motion deblurring techniques. Among them, blind decon volution is 
widely adopted to enhance a single blurred image, which may be applied under 
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different assumptions on the point spread function (PSF). Alternatively, several 
images with different blurring directions or an image sequence can be used, in 
more general situations, to estimate the PSF. In both cases, due to the 
discretization and quantization of images in both spatial and temporal coordinates, 
the PSF can not be reliably estimated, which produces a result inferior to the 
ground truth image (which is an image either taken with a camera on a tripod or of 
a static scene with correct exposure). A hybrid imaging system consisting of a 
primary (high spatial resolution) detector and a secondary (high temporal 
resolution) detector has also been proposed. The secondary detector provides more 
accurate motion information to estimate the PSF; thus, making deblurring possible 
even under long exposure. However, this technique needs additional hardware 
support, and the deblurred images are still not visibly as good as the ground truth 
in detail. 

[0010] Accordingly, the present solutions fail to provide sufficient image 

quality. 
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SUMMARY 



[0011] Techniques are disclosed to improve quality of images that may be 
blurred or underexposed (e.g., because of camera shake, taken in dim lighting 
conditions, or taken of high action scenes). The techniques may be implemented in 
a digital camera, digital video camera, or a digital camera capable of capturing 
video. 

[0012] In one described implementation, a digital camera includes an image 

sensor, a storage device, and a processing unit. The image sensor captures two 
images from a same scene which are stored on the storage device. The processing 
unit enhances the captured images with luminance correction. 

[0013] In another described implementation, the luminance correction 

includes determining a spatial coherence and color statistics of the first and second 
images. The determined color statistics and spatial coherence is utilized to enhance 
an underexposed one of the first and second images. 
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BRIEF DESCRIPTION OF THE DRAWINGS 



[0014] The detailed description is described with reference to the 

accompanying figures. In the figures, the left-most digit(s) of a reference number 
identifies the figure in which the reference number first appears. The use of the 
same reference numbers in different figures indicates similar or identical items. 

[0015] Fig. 1 illustrates an exemplary digital camera configuration for 

taking multiple shoots that may be utilized to provide luminance correction. 

[0016] Fig. 2 illustrates an exemplary storage configuration for storing 

image data captured by the sensor shown in Fig. 1 . 

[0017] Fig. 3 illustrates an exemplary graph indicating how photon energy 

captured by the sensor of Fig. 1 may increase over time. 

[0018] Fig. 4 illustrates an exemplary method for providing luminance 

correction in digital cameras. 

[0019] Figs. 5 and 6 illustrate images taken of a same scene under dim 

lighting conditions. 

[0020] Fig. 7 illustrates an exemplary high-quality image where luminance 

correction is applied. 
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[0021] Fig. 8 illustrates an exemplary method for luminance correction. 

[0022] Fig. 9 illustrates an exemplary method for color histogram 

equalization which may be utilized in luminance correction. 

[0023] Fig. 10A shows a homogeneous region from an original image, 

while Fig. 10B shows the same region taken with motion blur. 

[0024] Fig. 11 A and 11B illustrate curves of pixel colors along one 

direction. 

[0025] Fig. 12 illustrates an exemplary spatial region matching method 

which may be utilized in luminance correction. 

[0026] Fig. 13 illustrates a sample segmentation of the image of Fig. 6 into 

regions. 

[0027] Fig. 14 illustrates an exemplary intermediate image resulting from 

eroding the regions within the image of Fig. 13. 

[0028] Fig. 15 illustrates the selected region centers of Figs. 13 and 14 as 

dots. 

[0029] Fig. 16 illustrates a sample input underexposed image. 
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[0030] Fig. 17 illustrates an exemplary image resulting from application of 

luminance correction to the image of Fig. 16. 

[0031] Fig. 18 illustrates an exemplary result when applying color transfer 
techniques to the image of Fig. 16. 

[0032] Fig. 19 illustrates an exemplary result with adaptive histogram 

equalization applied to the image of Fig. 16. 

[0033] Fig. 20 illustrates an exemplary result when Gamma correction of 

2.5 is applied to the image of Fig. 16. 

[0034] Fig. 21 illustrates an exemplary result with curve adjustment applied 

to the image of Fig. 16 in a photo editing program. 

[0035] Figs. 22-25 illustrate the ability of the methodologies described 

herein to optimally combine the color information of two input images with 
varying exposure periods. 

[0036] Figs. 26-28 illustrate results of an experiment with motion blur 

caused by movement of objects within a scene. 

[0037] Sample images associated with a high contrast scene implementation 

are shown in Figs. 29-32. 



Attorney's Docket No. MS1-2020US 

lee@hayes poc 50*32*9256 



8 



Patent Application 
Express Mail No. EV 436703109 US 



[0038] Fig. 33 illustrates a general computer environment, which can be 

used to implement the techniques described herein. 
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DETAILED DESCRIPTION 



[0039] The following disclosure describes techniques for improving the 

quality of images that may be blurred or underexposed (e.g., because of camera 
shake, taken in dim lighting conditions, or taken of high action scenes). Two 
pictures are taken of a same scene with different exposure intervals. Hence, one 
image can be underexposed and the other can be blurred. The information within 
these two images is used to provide a high-quality image of the scene without 
visible blurring or darkness. The two pictures may be taken within a short interval, 
for example, to ensure that the center of the images do not move significantly or to 
limit the affects of motion by the camera or movement of objects within the scene. 

[0040] The techniques may be readily extended to handle high contrast 

scenes to reveal fine details in saturated regions (as will be discussed with 
reference to Figs. 22-25). Furthermore, some of the techniques may be directly 
incorporated into a digital camera, digital video camera, or a digital camera 
capable of capturing video. For example, a digital camera may be configured to 
keep its shutter open while taking the two pictures (as will be further discussed 
with reference to Figs. 1 , 4, and 8). 

[0041] Digital Camera Configuration 

[0042] Fig. 1 illustrates an exemplary digital camera configuration 100 for 

taking multiple shoots that may be utilized to provide luminance correction. The 
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camera may be a stand-alone camera or a camera incorporated into another device 
(such as a personal digital assistant (PDA), a cell phone, and the like). The 
configuration 100 includes a lens 102 which is exposed to light rays. Multiple lens 
configurations may be utilized to capture the light rays such as different types of 
lenses (e.g., zoom, fish eye, wide angle, etc.). 

[0043] Also, the configuration may further include an optional shutter (not 

shown). The shutter may control exposure of a sensor 104 to the light rays entering 
through the lens 102. Accordingly, the shutter may be located between the sensor 
104 and the lens 102. The shutter may be activated by a button on the camera or 
remotely (e.g., by an infra red or radio frequency remote control). Two pictures 
may be taken by the camera by pressing a shutter button a single time (e.g., so that 
luminance correction may be applied to the images as will be further discussed 
herein, for example, with reference to Figs. 4 or 8). The sensor 104 may be a 
CMOS or a charge-coupled device (CCD). 

[0044] In an implementation, the sensor 104 may be activated (e.g., 

powered) as needed without requiring a physical barrier (such as the shutter). 
Moreover, a more simplified mechanism (such as a sensor cover) may be utilized 
to protect the lens 102 and the sensor 104 from environmental elements (e.g., 
strong sun rays, dust, water, humidity, and the like). 

[0045] The digital camera configuration 100 further includes a digital 

camera processing unit 106 that is coupled to the sensor 104. The processing unit 
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106 includes a processor (108) and storage (110 and 112) to receive, store, and/or 
process the images captured by the sensor 104 (as will be further discussed herein, 
e.g., with reference to Fig. 4). It is envisioned that multiple processors (108) may 
be utilized, for example, to provide speed improvements. Also, the processing unit 
(106) or processors (108) may be specially designed for imaging applications. 

[0046] As illustrated, the processing unit 106 may include non- volatile 

memory, such as read only memory (ROM) (110). In one implementation, the data 
stored on the ROM 110 is utilized to provide camera settings, boot information, 
and the like (e.g., during start-up or upon request). Instead of or in addition to the 
ROM 110, flash memory may be utilized to allow changes. Also, other forms of 
rewritable storage may be used such as electrically erasable programmable read- 
only memory (EEPROM). Furthermore, the storage 112 may contain data and/or 
program modules that are accessible to and/or presently operated on by the 
processor 108 as will be further discussed with reference to Fig. 2. 

[0047] The digital camera configuration 100 may include other 

removable/non-removable, volatile/non-volatile computer storage media (not 
shown). By way of example, a hard disk drive or other non-volatile magnetic 
media (e.g., a "floppy disk"), an optical disk drive (such as a compact disc ROM 
(CD-ROM) and/or digital versatile disk (DVD)), a tape (e.g., in case of digital 
video cameras), and the like may be utilized to provide storage to the digital 
camera configuration 100. 
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[0048] The digital camera configuration 100 may optionally include a 

luminance correction logic 114 to apply luminance correction to captured images 
as will be further discussed herein, for example, with reference to Figs. 2, 4, and 8. 
The luminance correction logic 114 may be implemented as an application- 
specific integrated circuit (ASIC), a programmable logic array (PLA), and the like. 

[0049] In an implementation, the digital camera configuration 100 may 

utilize one or more external hardware facilities (such the computing environment 
discussed with reference to Fig. 33) to process and/or store data instead of or in 
addition to the digital camera processing unit 106. In such an implementation, the 
digital camera configuration 100 may also be controlled by the external hardware 
facility. This setup may free the photographer from manually changing the camera 
parameters between shots to enable the photographer to focus on shooting the best 
pictures. Data may be exchanged with the external hardware facility through a 
wired connection (e.g., universal serial bus (USB), fire wire (e.g., IEEE 1394), and 
the like) and/or wireless connection (e.g., 802.11 (and its varieties), cellular 
network, radio frequency, etc.). 

[0050] Fig. 2 illustrates an exemplary storage 112 for storing image data 

captured by the sensor 104 of Fig. 1. The storage 112 includes at least two buffers 
or caches (202 and 204) for storing two successive shots captured (e.g., by the 
digital camera discussed with reference to Fig. 1). Alternatively, the buffers (202 
and 204) may be implemented logically (i.e., physically located on a same 
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memory device but logically separate). The processor 108 of Fig. 1 may have 
access to the buffers (202 and 204) to permit processing of the captured images, 
for example, for luminance correction (see, e.g., Fig. 4). 

[0051] The storage 112 may include additional buffers (206), for example, 

to provide data caching while manipulating the captured images. Additional 
buffers may also be utilized to store newly captured images while the previously 
captured images are manipulated. Also, the storage 112 may optionally store 
luminance correction code or instructions 208 (e.g., in a buffer or a separate 
memory device). In such an implementation, the processor 108 may access the 
luminance correction code 208 to perform luminance correction (see, e.g., Figs. 4 
and 8). 

[0052] Fig. 3 illustrates an exemplary graph 300 indicating how photon 

energy captured by the sensor 104 of Fig. 1 may increase over time (e.g., from Tl 
to T2). For example, at time Tl, the sensor image may be underexposed in dim 
lighting conditions (as discussed with reference to Fig. 5). Also, at time T2 (e.g., Vi 
second after time Tl), the sensor image may be blurred (e.g., due to motion by 
camera, objects within the scene, etc.) as discussed with reference to Fig. 6. This 
difference in the exposure may be used in one implementation to provide 
luminance correction as will be further discussed with reference to Fig. 4 below. 
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[0053] Luminance Correction in Digital Cameras 

[0054] Fig. 4 illustrates an exemplary method 400 for providing luminance 

correction in digital cameras. The method 400 may be implemented as software, 
hardware, firmware, or combinations thereof within a digital camera such as that 
discussed with reference to Fig. 1 . 

[0055] Upon receiving a command to capture images (402), e.g., by 

pressing a button on a stand-alone digital camera or a camera incorporated into 
another device (such as a PDA, a cell phone, and the like), the camera sensor (e.g., 
104 of Fig. 1) is exposed to light rays. A first image is captured at a time Tl (406). 
The time Tl may be that discussed with reference to graph 300 of Fig. 3. 
Accordingly, the first image may be underexposed (e.g., I L of Fig. 5). 

[0056] A second image is then captured at a time T2 (406) of the same 

scene. The time T2 may be that discussed with reference to graph 300 of Fig. 3. 
Accordingly, the second image may be blurred (e.g., I H of Fig. 6). In an 
implementation, an exposure bracketing feature of a digital camera may be utilized 
for capturing the two images. Moreover, some cameras already include exposure 
bracketing (e.g., Canon G-model and some Nikon Coolpix model digital cameras) 
which takes multiple pictures at different shutter speeds by pressing the shutter 
button a single time. However, using the present built-in camera functionality has 
some limitations. Namely, it does not operate in manual mode, and the difference 
of shutter speeds can be limited. 
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[0057] Luminance correction is then applied to the captured images (408) 

such as discussed herein (see, e.g., Fig. 8) to provide a high quality image (e.g., I c 
of Fig. 7). 

[0058] In digital cameras with shutters, the shutter is left open to capture 

both an underexposed and a blurred image of the same scene. Such an 
implementation may ensure that any motion (e.g., from camera or objects within 
the scene) is limited. Also, leaving the shutter open may also improve speed (i.e., 
no lag time for intermediate shutter operation) and/or quietness (e.g., for taking 
nature shots). 

[0059] In one implementation, software for performing luminance 

correction (408) may be provided through a general-purpose computer (such as 
that discussed with reference to Fig. 33). The computer may be directly coupled to 
the camera (e.g., Fig. 1) or a memory card from the camera may be later linked to 
the computer to make the images available for processing. 

[0060] In a further implementation, the method 400 is implemented in a 

digital video camera (or a digital camera capable of capturing video). The video 
camera may be high-speed camera (e.g., capturing about 45 to 60 frames per 
second). The sensor of the camera (e.g., 104 of Fig. 1) may be reconfigured for 
every frame/field to adjust integration time, exposure timing, and/or analog to 
digital conversion (ADC) gain. Alternatively, two or more sensors with different 
settings may be utilized to capture successive images. 
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[0061] Sample Images 



[0062] Figs. 5 and 6 illustrate images taken of a same scene under dim 

lighting conditions. As illustrated, the image of Fig. 5 (I L ) is underexposed and the 
image of Fig. 6 (I H ) is blurred. The images of Figs. 5 and 6 have different exposure 
intervals. As illustrated, the image of Fig. 5 (I L ) has a longer exposure time than 
the image of Fig. 6 (I H ). Techniques discussed herein can utilize the images shown 
in Figs. 5 and 6 to construct a high-quality image (I c ) such as shown in Fig. 7, e.g., 
by applying luminance correction further discussed below with reference to Fig. 8. 

[0063] Image Acquisition 

[0064] In one implementation, to exploit the tradeoff between the exposure 

time and the blurring degree of the captured images, the two input images may be 
taken using the same capture device (e.g., a camera) with the following exposure 
settings: 

• One image (e.g., I L of Fig. 5) is taken with exposure time 
around the safe shutter speed, producing an underexposed 
image where motion blur is largely reduced. Since this image 
(I L ) is too dark, the colors in the image are not acceptable. 

• The other image (e.g., I H of Fig. 6) is an image acquired under 

an extended exposure time. The color and brightness of this 
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image is acceptable. However, it is motion blurred because of 
camera shaking or moving objects in the scene. 

[0065] In situations where movement of the scene (or objects within the 

scene) and/or capturing device (e.g., handheld camera without a tripod) is 
possible, the two pictures may be taken within a short interval. If the time lapse is 
kept as short as possible, the differences between the two images are minimized 
and/or the regional match of the positions of each pixel is maximized. 

[0066] Luminance Correction 

[0067] Fig. 8 illustrates an exemplary method 800 for luminance correction. 

After two images of the same scene are provided (802), such as discussed with 
reference to Figs. 5 and 6 (e.g., I L and I H ), the color statistics (804) and spatial 
coherence (806) of the images are determined (as will be described in more detail 
below in sections with similar titles). As discussed with reference to Figs. 5 and 6, 
I L and I H are two images of the same scene with different exposure intervals. 
Therefore, they are related not only by the color statistics, but also by the 
corresponding spatial coherence. 

[0068] The color statistics and spatial coherence information is utilized 

(808) to enhance the underexposed image (e.g., I L of Fig. 5) in color space to 
provide a normally exposed high quality image (e.g., I c of Fig. 7). More 
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specifically, the stage 808 utilizes a color mapping approach in one 
implementation. The color mapping is constrained by spatial details determined 
from the underexposed image and, thus, differs from and improves on previous 
pure color transfer techniques. As will be further discussed in more detail below, 
by properly formulating color statistics and spatial constraints, and incorporating 
them into a Bayesian framework, a maximum a posterior (MAP) solution provides 
an optimal color mapping function in the color space that preserves structural 
details while enhancing pixel colors simultaneously. 

[0069] Furthermore, the method 800 may deal with camera shake and object 

movement at the same time, and in a unified framework. Moreover, change of 
object topology or object deformation can also be handled, which is difficult for 
most deblurring methods, since different parts of the object have different PSFs. In 
addition, by slightly modifying one constraint (as will be further discussed under 
"color statistics in high contrast scenes"), the method 800 can be extended to deal 
with high contrast scenes and produce images with captured fine details in 
highlight or saturated areas. 

[0070] In an implementation, the method 800 is readily applied in low light 

conditions where artificial light sources such as flashes are not present (e.g., 

because of power consumption limitations such as in compact cameras, PDAs, 

cameras integrated into watches, pens, etc.), undesired (such as capturing images 

of biological matter where presence of light may result in undesired biological 
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phenomena), or otherwise impractical (e.g., due to distances or environmental 
conditions as in traffic control cameras). Examples where artificial lighting is 
undesired may include cameras attached to microscopes, electronic microscopes, 
etc. 

[0071] Relationship Between I I and l H 

[0072] As discussed with reference to Figs. 5-6, I L and I H are two images of 

the same scene with different exposure intervals. Therefore, they are related not 
only by the color statistics, but also by the corresponding spatial coherence. Their 
relationship may be translated into constraints for inferring a color mapping 
function in a Bayesian framework. 

[0073] In an implementation, the underexposed image I L can be regarded as 

a sensing component in normally exposed image I H in the temporal coordinates. 
This makes it possible to reasonably model the camera or scene (or scene object) 
movement during the exposure time, and constrain the mapping process which 
will be further described in the next sections. 

[0074] Color Statistics 

[0075] In RGB (red, green, and blue) color space, important color statistics 

can often be revealed through the shape of a color histogram. A histogram is 
generally a representation of a frequency distribution by means of rectangles or 
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bars whose widths represent class intervals and whose areas are proportional to the 
corresponding frequencies. Thus, the histogram can be used to establish an 
explicate connection between I H and I L . Moreover, since high irradiance generates 
brighter pixels, the color statistics in I L and I H can be matched in order from lower 
to higher in pixel intensity values. Accordingly, the histogram of I L (A^)may be 

reshaped such that: 

g(K)=h lH (i) 

[0076] In (1), g( ) is the transformation function performed on each color 

value in the histogram, and h lH is the histogram of I H . A common way to estimate 

g( ) is adaptive histogram equalization, which normally modifies the dynamic 
range and contrasts of a image according to a destination curve. 

[0077] This histogram equalization may not produce satisfactory results in 

some situations though. More specifically, the quantized 256 (single byte 
accuracy) colors in each channel may not be sufficient to accurately model the 
variety of histogram shapes. 

[0078] Fig. 9 illustrates an exemplary method 900 for color histogram 

equalization which may be utilized in luminance correction. More specifically, the 
method 900 may be utilized to optimally estimate the transformation function. 
First, the image from RGB space is transferred to a perception-based color space 
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lap (902), where the / is the achromatic channel, and a and fi contain the 
chromaticity value. In this way, the image is transformed to a more discrete space 
with known phosphor chromaticity. 

[0079] The color distributions in the new color space are clustered into 

65,536 (double byte precision) portions (904). Histogram equalization is then 
performed in the new color space (906). The result of the histogram equalization 
(906) is transferred back to the RGB space (908). 

[0080] By performing this transformed histogram equalization on the two 

images (e.g., Figs. 5-6), the two images may be related entirely in their color 
space. However, the color statistics is largely dependent on the image quality of 
the camera. For example, if the darker image contains a large amount of noise, the 
contaminated information may need to be treated first (e.g., by filtering the 
image). 

[0081] Spatial Constraint 

[0082] The color statistics described above does not consider any temporal 

coherence between I H and I L . However, since the two images are taken of the same 
scene, there is a strong spatial constraint between I H and I L . 

[0083] In a situation where a region contains similar color pixels, Fig. 10A 

shows a homogeneous region from the original image, while Fig. 10B shows the 
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same region taken with motion blur. The dots 1002 mark the region centers. The 
corresponding curves of Fig. 11A and 11B illustrate pixel colors along one 
direction. From Figs. 10A, 10B, 11 A, and 11B, it can be observed that the color 
toward the center of the region is less affected by blurring, given that the region 
area is sufficiently large and homogeneous. Additionally, the consistency of colors 
in the region allows matching of the color of central pixels. 

[0084] Fig. 12 illustrates an exemplary spatial region matching method 

1200 which may be utilized in luminance correction. The method 1200 may be 
used to select matching seeds in I H and I L . The blurred image (e.g., I H of Fig. 6) is 
segmented into regions such that each region R m {I H ) contains similar colors 
(1202). A sample segmentation of Fig. 6 into regions is shown in Fig. 13. 

[0085] To sort the regions according to the homogeneity and size, each 

region R m (I H ) is eroded (1204) and the number of iterations to completely erode 
each region and the region centers which are the last few pixels in the eroding 
process for each region are determined (1206). The same morphological eroding 
operation may be performed for each region R m {In) in one implementation. Fig. 14 
illustrates an exemplary intermediate image resulting from eroding the regions 
within the image of Fig. 13. 

[0086] The iteration numbers are sorted in descending order and the first M 

regions are selected as the most possible candidates for region matching (1208). 
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As a result, the positions of these region centers are selected as matching 
positions. From the images I H and I L , pixel pairs {c" y c™} in the matching position 

are selected (1210) and the value for each c m is calculated as a Gaussian average of 
the colors of neighboring pixels (1212), where the variance is proportional to the 
iteration numbers. The selected region centers are illustrated in Fig. 15 as dots 
(1502). As illustrated, the dots (1502) are located in the largest and most 
homogeneous M regions. 

[0087] The matching process (1200) implies that an ideal color mapping 

function should be able to transform some matching seeds colors in I L to those in 
I H . In the next section, a Bayesian framework is described which incorporates the 
two constraints (color and spatial) into consideration, so as to infer a constrained 
mapping function. 

[0088] Constrained Mapping Function 

[0089] The color mapping function may be defined as /(/,-) = /,-, where /, 

and l\ are color values in two sets, respectively. Accordingly, the resulting image 
I c is built by applying^ ) to the underexposed image I L \ Idpc, y) =J[I L (x t y)), where 
I k (x, y) is pixel values in image I k . Note that the form of/ ) is constrained by both 
I L and I H . 



Attorney's Docket No. MS1-2020US 

lee® hay es \*c 509-324-9256 



24 



Patent Application 
Express Mail No. EV 436703109 US 



[0090] In Bayesian framework, one maximizes the a posterior probability 

(MAP) to infer f* given the observations from I L and I H \ 



/^ = argmax/7(/|/ L ,/ w ) (2) 

[0091] In the previous sections, two kinds of connections between I L and I H 

were observed. One is color statistics which can be described by two histograms 
h lL and h Ju of I L and I Hy respectively. The other is the region matching constraint 

which can be represented by a number of M corresponding matching color seeds 
{ c L> c H}m=\ between I L and I H . These relationships may be regarded as constraints 
and equation (2) may be rewritten as: 

f* = arg max p(f\ h h , h, H , {c m L9 c m H }% =] ) 

- arg max p(h ti , h Su , {c m L9 c m H }^ \f) P(0 (3) 

[0092] The next sections defines the likelihood p(h h , h Iff , {c m L ,c m H } M m=x \f) and 

the prior /?(/). 
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[0093] Likelihood 



[0094] Since global matching is performed in a discrete color space, / is 

approximated by a set of discrete values / '= {fl, fl, fi,..., fN}, where N is the 
total number of bins in the color space. Hence, the likelihood in equation (3) can 
be factorized under the independent and identically distributed (IID) assumption: 

/>( V h lH , {c m L ,c"X =l \f) = flp(g(lMc?rf}\fi) (4) 

[0095] In equation (4), is a function to transform h h to h ffj at color 

value //. The c~' is the most similar color to /, in color seeds set {c™}^ } , and c'J is 
the corresponding color of c~' in color seed pairs. 

[0096] According to the analysis in the previous sections, g(l t ) and {c~',c~'} 

are two constraint factors for each f h Both of their properties should be maintained 
on the mapping function. As a consequence, the two constraints may be balanced 
and the likelihood may be modeled as follows: 



P(g(ld, {c^c-^lfi oc exp 



2a) 



(5) 



[0097] In equation (5), the scale a weights the two constraints, and a) is a 

variance to model the uncertainty of the two kinds of constraints. As the value of a 
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grows, the confidence of the matching seed pairs drops. The a may be related to 
the following factors: 



• The distance ||/, - c L '\\. A large distance indicates a weak 

region matching constraint, which makes a approach 1. 
Hence, the a is inversely proportional to this distance. 

• The uncertainty of correspondence in matching color pair 
{c~',c~'} . As depicted in the previous sections, the larger the 

matching region size is, the larger confidence one can get 
from the region center for the matching colors. Hence, the 
uncertainty a c may be defined to be proportional to the region 
size for each matching color. Combining these two factors, 
the a may be defined as: 



a = exp 



<y]\\i-c-,:\t 

2p 2 



(6) 



[0098] In equation (6), fi is the scale parameter to control the influence of a. 

[0099] Prior 

[00100] As a prior, the monotonic constraint may be enforced onf( ), which 

maintains the structural details in I L . In addition, to avoid abrupt change of the 
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color mapping for neighboring colors, it may be required that^( ) be smooth in its 
shape in an implementation. In another implementation, the second derivative of / 
may be minimized as follows: 



p{f)cc exp 



( 1 ^ 



J(/7 



oc exp 



v 2<T f 



(7) 



[00101] In equation (7), the aj is the variance to control the smoothness of / 
[00102] MAP Solution 

[00103] Combining the log likelihood of equation (4) and the log prior in 
equation (7), the optimization problem may be solved by minimizing the following 
log posterior function: 

E(f) = -Xlogp(g(/,),K',c w '} |/ ( ) -log/*/) (8) 

i 

[00104] In equation (8), the E(J) is a quadratic objective function. Therefore, 
the global optimal mapping function J[ ) can be obtained by the singular value 
decomposition (SVD). Although the monotonic constraint is not enforced 
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explicitly in equation (7), the smoothness constraint is sufficient to construct the 
final monotonic fin an implementation. 

[00105] Other Sample Results 

[00106] The techniques described herein are applied to difficult scenarios to 
show the efficacy of the approach. The results are classified into different groups 
as follows: Fig. 16 illustrates a sample input underexposed image; Fig. 17 
illustrates an exemplary image resulting from application of the luminance 
correction of the present application to the image of Fig. 16; Fig. 18 illustrates an 
exemplary result when applying color transfer techniques; Fig. 19 illustrates an 
exemplary result with adaptive histogram equalization; Fig. 20 illustrates an 
exemplary result when Gamma correction of 2.5 is applied; and Fig. 21 illustrates 
an exemplary result with curve adjustment applied in a photo editing program. As 
can be seen, better visual quality and more details are achieved by using the 
constraints described herein (i.e., Fig. 17). 

[00107] Also, the two constraints described herein (spatial and color) are 
both beneficial in an implementation. They optimize the solution in two different 
aspects. Therefore, the combination and balance of these constraints may 
guarantee the visual correctness of the methodology described herein in one 
implementation. 
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[00108] 



Motion Blur Caused by Hand-Held Camera 



[00109] The rock example of Figs. 22-25 illustrates the ability of the 
methodologies described herein to optimally combine the color information of the 
two input images (i.e., those of Figs. 22 and 23). The histogram of each image is 
shown in the lower left side of each figure. Unlike other deblurring methods, the 
resulting edges are crisp and clear. The two input images are taken with shutter 
speeds 1/40 s and 1/3 s, respectively. Figs. 24 and 25 are the color mapped image 
(I c ) and ground truth with their corresponding histograms, respectively. The 
ground truth image (Fig. 25) is taken by using a tripod. Note that the images are 
visually and statistically close in Figs. 24 and 25. 

[00110] Motion Blur Caused by Movement of Objects 

[00111] In an implementation, the techniques discussed herein can easily 
solve object movement or deformation problems (e.g., if the object movement is 
too fast in normal exposure interval). Figs. 26-28 illustrate results of an 
experiment. As illustrated, portions of Figs. 27 and 28 are enlarged for ease of 
reference. The input normal exposure image is locally blurred (Fig. 27), i.e., PSF 
has no uniform representation in the whole image, which easily makes 
deconvolving methods fail. Using the techniques discussed herein (and the 
underexposed image of Fig. 26), the camera shutter speed may be reduced by four 
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stops. As a result, a high quality image (I c ) with largely reduced blurring effect can 
be generated (Fig. 28). 

[00112] Color Statistics in High Contrast Scenes 

[00113] Where the images are taken of a high contrast scene, bright regions 
will become saturated in I H . Histogram equalization faithfully transfers colors 
from I L to I H , including the saturated area, which not only degrades the spatial 
detail in the highlight region, but also generates abrupt changes in the image color 
space. 

[00114] To solve this problem, the color mapping function g{ ) described in 
the previous sections may be modified to cover a larger range. In one 
implementation, a color transfer technique may be utilized to improve the image 
quality in high contrast situations. This technique also operates on an image 
histogram, which transfers the color from the source image to the target by 
matching the mean and standard deviation for each channel. It has no limit on the 
maximum value of the transferred color since the process is a Gaussian matching. 

[00115] In an implementation, all non-saturated pixels in I H are used for 
color transfer to I L . After applying the color transfer technique, the mapping result 
of I L exceeds the color depth (that is, above 255), and extends the saturated pixels 
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to larger color values. Hence, a higher range image is constructed to reveal details 
in both bright and dark regions. 

[00116] Sample images associated with such an implementation are shown in 
Figs. 29-32. As illustrated, portions of Figs. 29-32 are enlarged for ease of 
reference. Figs. 29 and 30 illustrate the input images (I H and I L , respectively). The 
image of Fig. 31 is reconstructed by setting g{ ) as the original histogram 
equalization function. Fig. 32 is a result with enhanced colors and details by 
modifying g( ) to use the color transfer method. Tone mapping is also performed to 
present the image illustrated in Fig. 32. 

[001 17] General Computer Environment 

[00118] Fig. 33 illustrates a general computer environment 3300, which can 
be used to implement the techniques described herein. For example, the computer 
environment 3300 may be utilized to run the software program that controls an 
image capture device (such as a camera). The computer environment 3300 is only 
one example of a computing environment and is not intended to suggest any 
limitation as to the scope of use or functionality of the computer and network 
architectures. Neither should the computer environment 3300 be interpreted as 
having any dependency or requirement relating to any one or a combination of 
components illustrated in the exemplary computer environment 3300. 
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[00119] Computer environment 3300 includes a general-purpose computing 
device in the form of a computer 3302. The components of computer 3302 can 
include, but are not limited to, one or more processors or processing units 3304 
(optionally including a cryptographic processor or co-processor), a system 
memory 3306, and a system bus 3308 that couples various system components 
including the processor 3304 to the system memory 3306. 

[00120] The system bus 3308 represents one or more of any of several types 
of bus structures, including a memory bus or memory controller, a peripheral bus, 
an accelerated graphics port, and a processor or local bus using any of a variety of 
bus architectures. By way of example, such architectures can include an Industry 
Standard Architecture (ISA) bus, a Micro Channel Architecture (MCA) bus, an 
Enhanced ISA (EISA) bus, a Video Electronics Standards Association (VESA) 
local bus, and a Peripheral Component Interconnects (PCI) bus also known as a 
Mezzanine bus. 

[00121] Computer 3302 typically includes a variety of computer-readable 
media. Such media can be any available media that is accessible by computer 3302 
and includes both volatile and non-volatile media, removable and non-removable 
media. 

[00122] The system memory 3306 includes computer-readable media in the 
form of volatile memory, such as random access memory (RAM) 3310, and/or 
non-volatile memory, such as ROM 3312. A basic input/output system (BIOS) 
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3314, containing the basic routines that help to transfer information between 
elements within computer 3302, such as during start-up, is stored in ROM 3312. 
RAM 3310 typically contains data and/or program modules that are immediately 
accessible to and/or presently operated on by the processing unit 3304. 

[00123] Computer 3302 may also include other removable/non-removable, 
volatile/non-volatile computer storage media. By way of example, Fig. 33 
illustrates a hard disk drive 3316 for reading from and writing to a non-removable, 
non-volatile magnetic media (not shown), a magnetic disk drive 3318 for reading 
from and writing to a removable, non-volatile magnetic disk 3320 (e.g., a "floppy 
disk"), and an optical disk drive 3322 for reading from and/or writing to a 
removable, non-volatile optical disk 3324 such as a CD-ROM, DVD, or other 
optical media. The hard disk drive 3316, magnetic disk drive 3318, and optical 
disk drive 3322 are each connected to the system bus 3308 by one or more data 
media interfaces 3326. Alternatively, the hard disk drive 3316, magnetic disk drive 
3318, and optical disk drive 3322 can be connected to the system bus 3308 by one 
or more interfaces (not shown). 

[00124] The disk drives and their associated computer-readable media 
provide non-volatile storage of computer-readable instructions, data structures, 
program modules, and other data for computer 3302. Although the example 
illustrates a hard disk 3316, a removable magnetic disk 3320, and a removable 
optical disk 3324, it is to be appreciated that other types of computer-readable 
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media which can store data that is accessible by a computer, such as magnetic 
cassettes or other magnetic storage devices, flash memory cards, CD-ROM, DVD 
or other optical storage, random access memories (RAM), read only memories 
(ROM), electrically erasable programmable read-only memory (EEPROM), and 
the like, can also be utilized to implement the exemplary computing system and 
environment. 

[00125] Any number of program modules can be stored on the hard disk 
3316, magnetic disk 3320, optical disk 3324, ROM 3312, and/or RAM 3310, 
including by way of example, an operating system 3326, one or more application 
programs 3328, other program modules 3330, and program data 3332. Each of 
such operating system 3326, one or more application programs 3328, other 
program modules 3330, and program data 3332 (or some combination thereof) 
may implement all or part of the resident components that support the distributed 
file system. 

[00126] A user can enter commands and information into computer 3302 via 
input devices such as a keyboard 3334 and a pointing device 3336 (e.g., a 
"mouse"). Other input devices 3338 (not shown specifically) may include a 
microphone, joystick, game pad, satellite dish, serial port, scanner, and/or the like. 
These and other input devices are connected to the processing unit 3304 via 
input/output interfaces 3340 that are coupled to the system bus 3308, but may be 
connected by other interface and bus structures, such as a parallel port, game port, 
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or a universal serial bus (USB). The USB port may be utilized to connect a camera 
or a flash card reader (such as discussed with reference to Fig. 29) to the computer 
environment 3300. 

[00127] A monitor 3342 or other type of display device can also be 
connected to the system bus 3308 via an interface, such as a video adapter 3344. 
In addition to the monitor 3342, other output peripheral devices can include 
components such as speakers (not shown) and a printer 3346 which can be 
connected to computer 3302 via the input/output interfaces 3340. 

[00128] Computer 3302 can operate in a networked environment using 
logical connections to one or more remote computers, such as a remote computing 
device 3348. By way of example, the remote computing device 3348 can be a 
personal computer, portable computer, a server, a router, a network computer, a 
peer device or other common network node, game console, and the like. The 
remote computing device 3348 is illustrated as a portable computer that can 
include many or all of the elements and features described herein relative to 
computer 3302. 

[00129] Logical connections between computer 3302 and the remote 
computer 3348 are depicted as a local area network (LAN) 3350 and a general 
wide area network (WAN) 3352. Such networking environments are commonplace 
in offices, enterprise- wide computer networks, intranets, and the Internet. 
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[00130] When implemented in a LAN networking environment, the 
computer 3302 is connected to a local network 3350 via a network interface or 
adapter 3354. When implemented in a WAN networking environment, the 
computer 3302 typically includes a modem 3356 or other means for establishing 
communications over the wide network 3352. The modem 3356, which can be 
internal or external to computer 3302, can be connected to the system bus 3308 via 
the input/output interfaces 3340 or other appropriate mechanisms. It is to be 
appreciated that the illustrated network connections are exemplary and that other 
means of establishing communication link(s) between the computers 3302 and 
3348 can be employed. 

[00131] In a networked environment, such as that illustrated with computing 
environment 3300, program modules depicted relative to the computer 3302, or 
portions thereof, may be stored in a remote memory storage device. By way of 
example, remote application programs 3358 reside on a memory device of remote 
computer 3348. For purposes of illustration, application programs and other 
executable program components such as the operating system are illustrated herein 
as discrete blocks, although it is recognized that such programs and components 
reside at various times in different storage components of the computing device 
3302, and are executed by the data processor(s) of the computer. 

[00132] Various modules and techniques may be described herein in the 
general context of computer-executable instructions, such as program modules, 
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executed by one or more computers or other devices. Generally, program modules 
include routines, programs, objects, components, data structures, etc. that perform 
particular tasks or implement particular abstract data types. Typically, the 
functionality of the program modules may be combined or distributed as desired in 
various implementations. 

[00133] An implementation of these modules and techniques may be stored 
on or transmitted across some form of computer-readable media. Computer- 
readable media can be any available media that can be accessed by a computer. By 
way of example, and not limitation, computer-readable media may include 
"computer storage media" and "communications media." 

[00134] "Computer storage media" includes volatile and non-volatile, 
removable and non-removable media implemented in any method or technology 
for storage of information such as computer-readable instructions, data structures, 
program modules, or other data. Computer storage media includes, but is not 
limited to, RAM, ROM, EEPROM, flash memory or other memory technology, 
CD-ROM, DVDs, or other optical storage, magnetic cassettes, magnetic tape, 
magnetic disk storage or other magnetic storage devices, or any other medium 
which can be used to store the desired information and which can be accessed by a 
computer. 

[00135] "Communication media" typically includes computer-readable 
instructions, data structures, program modules, or other data in a modulated data 
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signal, such as carrier wave or other transport mechanism. Communication media 
also includes any information delivery media. The term "modulated data signal" 
means a signal that has one or more of its characteristics set or changed in such a 
manner as to encode information in the signal. By way of example, and not 
limitation, communication media includes wired media such as a wired network or 
direct-wired connection, and wireless media such as acoustic, RF, infrared, and 
other wireless media. Combinations of any of the above are also included within 
the scope of computer-readable media. 

[00136] Conclusion 

[00137] Although the invention has been described in language specific to 
structural features and/or methodological acts, it is to be understood that the 
invention defined in the appended claims is not necessarily limited to the specific 
features or acts described. Rather, the specific features and acts are disclosed as 
exemplary forms of implementing the claimed invention. For example, the 
luminance correction techniques discussed herein may be readily applied to non- 
color images (e.g., grayscale images). 
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